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This Brief 


The Student Achievement Initiative (SAI), adopted by the Washington State 
Board for Community and Technical Colleges in 2007, is one of a growing 
number of performance funding programs that have been dubbed “perfor- 
mance funding 2.0.” Unlike previous performance funding models, the SAI 
rewards colleges for students’ intermediate achievements along the pathway 
toward college completion as well as for completion itself. In addition to mo- 
tivating colleges through funding, the SAI gives colleges data to help them 
understand where, along their pathways through college, students are strug- 
gling and what changes might improve their forward momentum. 

Washington State’s experience with the SAI offers clear lessons about the 
process of developing, implementing, and monitoring a performance fund- 
ing system. This brief uses findings from a three-year evaluation of the SAI 
to draw lessons for leaders in other states seeking to design effective perfor- 
mance funding systems. On the basis of this work, we offer the following 
suggested design principles. 


* Measure what matters most (and validate that you 
have done so). States should carefully measure the 
outcomes they consider most important, even if 
new measurement approaches are needed. More- 
over, they should continually monitor the validity 
of the measures they are using and their alignment 
with goals, as goals may shift over time. 

* One-size metrics do not fit all stakeholders. 

State policymakers need metrics that speak to 
publicly valued outcomes. Colleges require a dif- 
ferent, more detailed set of metrics that will help 
faculty and staff understand which campus prac- 
tices and policies may need to change in order to 
achieve better results. State and system agencies 
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should work with colleges to develop and use 
metrics that help to inform institutional im- 
provement and that also align with policymak- 
ers’ metrics so that institutions can ultimately see 
the impact of their reforms in the metrics used for 
public accountability and funding. 

* Everyone can’t win — but guard against un- 
expected consequences. Many performance 
funding systems try to minimize competition 
by banking on new funds and keeping the stakes 
small. A better approach is to use fiscal incentives 
to motivate change, phase in consequences to 
allow for learning and adjustments, and provide 
technical assistance for colleges that need it. At 
the same time, colleges should not be disadvan- 
taged for serving vulnerable populations. 

* Measure improvements in performance rather 
than relative performance. Because college 
performance is determined in part by the char- 
acteristics of the students served and the mix 
of programs offered, a college’s performance is 
best measured against its own historical baseline 
rather than against the performance of other col- 
leges. Measuring improvements in performance 
is the fairest way to deal with the problem of 
trying to compare the performance of colleges 
that serve more disadvantaged students with the 
performance of those that serve more advantaged 
students. It also helps colleges to focus on their 
own performance and what they need to change 
to improve outcomes for the students they serve. 


* Performance is a baseline expectation. Fiscal 
incentives for performance should be structured 
as part of base budget funding. A base funding 
formula that values student progress, comple- 
tion, and other desired outcomes as much as en- 
rollment would give colleges their full allocation 
in a timely and predictable way and would allow 
for large enough amounts to be allocated on the 
basis of performance — increasing the motivation 
for systemic rather than piecemeal change. 

* It’s about mission, not rewards. To the extent 
that the conversation about performance funding 
can shift in focus from additional rewards and 
discrete practices to base funding and systemic 
institutional improvement, the power of the 
fiscal incentives may be better harnessed to the 
benefit of student success. 

* Performance is everyone’s business. The 

process of institutional improvement requires 
broad participation. Faculty and staff must 
be actively engaged in interpreting data and 
designing improvement strategies, and college 
leaders must support them in this work. Institu- 
tional research staff should coordinate a process 
across the college to monitor student progress 
and institutional goal achievement and to link 
patterns of progress to institutional practices, 
policies, and resource allocation. States can help 
build the capacity of colleges to do this in ways 
that lead to improved institutional performance 
and student success. 
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Introduction: Washington’s 
Student Achievement Initiative 
as a Model for Other States 


As the nation’s colleges and universities struggle with declining public funding, growing enroll- 
ments, and calls for improved completion rates, state lawmakers are once again looking at perfor- 
mance-based funding as a way to generate a better return on public investments in higher education. 
This renewed interest in performance funding derives from observations that enrollment-based 
funding has led institutions to focus on maximizing enrollment and thus pay insufficient attention 
to student outcomes. By tying institutional funding to completion and other desired outcomes, the 
expectation is that colleges will have added motivation to identify and implement better ways to ac- 
complish those goals. 

Earlier models of performance-based funding encountered much resistance from educators and were 
often short-lived. 1 Designed mostly to reward degree completion, often with arbitrary targets and 
few dollars, they were rightly criticized for imposing standards and metrics on institutions that did 
not accurately capture their missions and complexities. Particularly for community colleges, which 
aim to provide “open access” to higher education, completion rates provide an incomplete measure 
of their effectiveness in achieving their mission. Thus, with performance -based funding, colleges 
could in effect be penalized for opening their doors to disadvantaged students, who are more expen- 
sive to serve and less likely to complete a program than students with more privileged backgrounds. 

The Washington State Student Achievement Initiative (SAI), adopted in 2007 by the State Board for 
Community and Technical Colleges, 2 is one of the growing number of performance funding pro- 
grams that have been dubbed “performance funding 2.0,” signaling their attempt to overcome the 
shortcomings of earlier models. 3 Two innovations in particular distinguish the SAI from previous 
performance funding models. First, the SAI measures and rewards colleges for students’ intermediate 
achievements along the pathway toward completion in addition to rewarding them for completion. 
The SAI’s performance metrics incorporate measures of achievement for students starting in adult 
basic skills or college remediation, so that colleges are not at a disadvantage for serving disadvantaged 
students. Under the SAI, colleges are awarded “achievement points” when students attain educa- 
tional milestones along the path to a degree (see table, “SAI Achievement Points”). Viewing student 
success as a continuous rather than discrete outcome is now widely accepted as the basis for more 
complete and more accurate assessment of community college performance. This model has been 
adopted in various forms by other states. 

The second innovative feature of the SAI is that it gives colleges data to help them understand where, 
along their pathways through college, students are struggling and what changes might improve 
their forward momentum. Colleges often lack sufficient information about what works to promote 
student success. Therefore, in addition to motivating colleges through funding, the SAI was designed 
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SAI Achievement Points 4 

Milestone Area 

Achievement Points 

Building toward college-level 
skills 

• Basic skills gains — increase in skill level based on a standardized test 

• Passing a pre-college writing or math course that would qualify 
student to advance to the next level 

First-year retention and 
progress 

• Earning 15 quarter credits of college-level coursework 

• Earning 30 quarter credits of college-level coursework 

Completing college-level math 

• Passing a math course required for either a technical or academic 
associate degree 

Completion 

• Completing a degree, an occupational certificate, or apprenticeship 
training 


to provide data to guide colleges in identifying barriers to student progression and taking steps to 
remove them in order to increase student completion rates. 

Understanding that many prior efforts at performance funding, including one in Washington, had 
failed to be embraced by college faculty and staff, the State Board used a highly inclusive process for 
designing the S AI, starting with the adoption of a set of principles developed in consultation with the 
college presidents. A system-wide task force then used the design principles to develop the operational 
details of the policy. 

This policy brief uses the findings from a three-year evaluation of the S AI 5 to draw lessons for leaders 
in other states seeking to design effective performance funding systems. The brief updates the findings 
described in a 201 1 brief, which offered a preliminary assessment of lessons drawn from the SAI. 6 In 
our earlier brief, we described the three main challenges faced by the State Board in developing the SAI: 

• designing a system of metrics that generates information useful to colleges; 

• designing a mechanism for linking performance to budgets that provides 
motivation for improvement; and 

• fostering the conditions by which the data and fiscal motivation produce 
systemic institutional change. 

These are the three main challenges that any state will face if it seeks to develop a performance funding 
system for its colleges. We address these same three issues in this brief with the benefit of two addition- 
al years to observe the implementation of the SAI, the completion of the three-year evaluation, and the 
State Board’s own review of and revisions to the SAI, which were completed in December 20127 

For each of the three challenges, we provide a critical assessment of how well the relevant Board- 
adopted design principles served the purpose of the SAI to improve system-wide student achieve- 
ment. We describe how the implementation of the design principles influenced the operation of 
the SAI, the changes the State Board made in December 201 2 to address any identified problems 
with initial implementation, and the strengths and weaknesses of the design principles adopted. We 
conclude the brief with some suggestions for principles to guide state policymakers in designing ef- 
fective performance funding systems. 
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Critical Assessment of the 

SAI Design Principles 

Developing Performance Metrics 

Impact of Design Principles on Implementation 

In designing the SAI, the State Board adopted several principles related to the measurement of per- 
formance under the SAI (see box, “Principles for Measurement”). To some extent, the SAI achieve- 
ment point framework for measuring performance satisfied these principles. In our fieldwork at 
colleges in the system, we found strong support for the way the SAI tracks progression across the full 
range of mission areas, including adult basic skills and college remediation. Those we interviewed 
also appreciated that the progression metrics were derived from research on milestones correlated 
with a higher likelihood of college completion. 

Principles for Measurement 

• Performance measures recognize students in all mission areas and reflect the needs of the 
diverse communities served by colleges. 

• Performance measures reflect incremental gains in students’ educational progress irrespective 
of mission area. 

• Measures are simple, understandable, and reliable and represent valid points in students’ edu- 
cational progress. 

• Measures focus on student achievement improvements that can be influenced by colleges. 


Most interviewees recognized that there is a tradeoff between the simplicity of a performance- 
based funding model and its ability to fully accommodate all important aspects of the college’s 
mission. However, there was a growing sense over the course of our three-year evaluation that the 
latter part of the SAI’s progression framework could be better rewarded, as there are considerably 
more opportunities to earn points in the pre-college portion of the point framework than in the 
college-level stages of student progression. 

State Board Revisions 

The revisions to the SAI adopted by the State Board in December 2012 sought to address this con- 
cern by adding points for students who earn 45 college credits (in addition to the 15- and 30- credit 
milestones, which were retained). The revisions also give more points for adult basic skills students 
who transition to college-level courses and for remedial students who pass a college-level course in 
the discipline in which they previously took remedial courses. 
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The Board’s revisions also attempted to address another concern with the metrics. Under the original 
framework, which based awards on changes in total points from the previous year, larger colleges 
generally outperformed smaller ones. It is true that larger colleges may need more resources to 
implement improvements. Yet the original framework allowed colleges to increase their points by 
increasing enrollment, even if the number of points per student did not increase. Indeed, our quanti- 
tative analysis showed a negative correlation between total points and points per student, the latter 
being a measure of effectiveness that is not influenced by college size. To balance the effect of size on 
awards, the Board changed to an award method based on a mix of metrics: 40 percent total points, 40 
percent points per student, and 20 percent completions. 

Our Assessment of the Principles 

Although the changes adopted by the Board addressed valid concerns from the colleges and will en- 
able the SAI to reward colleges more fairly, they made the points system more complex. College staff 
had already complained that they were not able to account for why their college performed better or 
worse than other colleges based on their points results, which made it difficult to identify ways to 
improve. Under the new framework, which adds points per student to the set of metrics, it will likely 
be even less clear how they can improve their performance. 

Another concern is the cross-sectional nature of the SAI metrics, which record the aggregate 
points earned by students in a college at a point in time, rather than the progress of students over 
time. In its revisions, the Board added two longitudinal measures: whether remedial students 
went on to take and pass a college-level “gatekeeper” course, and year-to-year persistence. Howev- 
er, most of the remaining metrics are cross-sectional in that they measure the number of points (or 
points per student) achieved by a college’s students in a given year. They do not indicate whether 
students are making progress from one milestone to the next. Indeed, our quantitative analysis 
found that, although colleges were gaining points, students’ forward movement or momentum 
was not increasing overall. 

There was growing awareness among staff at the colleges where we conducted our interviews that to 
identify strategies for improving student outcomes and to measure their impact would require data 
on cohorts of students tracked over time rather than cross-sectional measures. Members of the Re- 
search and Planning Commission, the association of institutional researchers in Washington State’s 
two-year colleges, in collaboration with the State Board staff, conducted longitudinal analyses using 
SAI data showing the highest outcomes achieved by students over two-, three-, four-, and five-year 
timeframes. Their development of this technique showed that although most of the SAI metrics are 
cross-sectional, the SAI data can be used for longitudinal cohort tracking. 

One reason that longitudinal tracking was not used in the original design of the SAI is that it takes 
a long time for many community college students to complete a degree, and thus there is a long lag 
time in measuring college performance. In our quantitative analysis of college performance using 
the SAI achievement points, we adapted the method developed by the Washington college institu- 
tional researchers and State Board staff to measure student progression from one year to the next. 

We looked at students grouped by their stage of achievement at the beginning of the base year (some 
who had just started and others who had earned achievement points in the past) and for each group 
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counted the number of students who earned additional points further along the pathway to comple- 
tion by the end of the next year. This approach provides a possible method to measure changes in 
student progression rates year-to-year without having to wait for the results of longitudinal analysis 
over a longer period. However they do it, the Washington community and technical colleges will 
need to continue using longitudinal cohort tracking in addition to tracking the cross-sectional SAI 
metrics if they are to use the SAI data to improve their performance. 

Linking Performance to Budgets 

Impact of Design Principles on Implementation 

The box titled “Principles for Incentive Funding” shows the original design principles adopted by 
the State Board that relate to the flow of funding under the SAI. The first two principles were made 
operational by the decision that dollar rewards to colleges would be based on each college’s change 
in total points as measured from a base year rather than on a measure of total points or one that 
compared performance across colleges. Beginning in 2008-09, colleges received a flat rate per point 
gained, based on performance in the prior year. Over the four years that performance funds have 
been allocated, the dollar amount per point gain has varied from $3 1 to $84 based on the available 
pool of SAI funds and the total of all colleges’ increase in points. Therefore, although the reward 
received by one college is not directly affected by its performance in comparison with other colleges, 
its award is influenced by the collective claim on the SAI funding pool by all colleges. 

Principles for Incentive Funding 

• Colleges are rewarded for improvements in student achievement. 

• Funding is structured so that colleges compete against themselves for continuous improvement 
rather than competing with each other. 

• Funding is stable, predictable, and cumulative over time. 

• New funds provide the greatest incentive. 

• Incentive funding rewards student success and becomes a resource for adopting and expanding 
practices leading to further success. 


The plan for honoring the third and fourth principles assumed that there would be an annual, 
increasing state allocation for the SAI — to be provided in addition to the system’s base general fund 
allocation. Each year’s SAI rewards were to be folded into colleges’ base allocations, thereby gradually 
increasing the amount of funds allocated to colleges on the basis of performance. The Board’s fund- 
ing model projected new dollars per point to increase from $30 in 2009 to $145 in 2013. There was 
no intention to reallocate any of the system’s base budget for the SAI or to modify how base budgets 
would be allocated, which was a “base-plus” method in which colleges received their prior year’s 
allocation adjusted for changes in system priorities and available funds. 

State budget realities thwarted this plan. Beginning in 201 1, the SAI was funded through a realloca- 
tion of funds from the system’s base budget, which has been reduced by over twenty percent since 
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the start of the recession in 2008. Consequently, the S AI funds originally billed as “new money” and 
as “rewards” became, for many colleges, only partial restoration of base funds lost, making it difficult 
to sustain the notion of the SAI as a reward system. 

The last principle for incentive funding was addressed, perhaps less explicitly than the others, 
through decisions about the amount of funds to be devoted to the SAI and the language used to de- 
scribe how these funds were to be used. Funding allocated for achievement points has never amount- 
ed to more than one percent of the college system’s budget. For the two years between the launch of 
the initiative and the initial allocation of funds based on performance, a flat amount was allocated to 
each college as “seed money” to support “student success strategies.” 8 The average allocation per col- 
lege was $ 5 2,000 in the first year and $ 67,000 in the second year. The table titled “Funding Awards 
History” shows the range of SAI awards to colleges over the next four years, during which awards 
reflected performance. Colleges responded to the relatively small amount of funding provided by 
enhancing existing practices or implementing relatively small-scale innovations rather than making 
systemic changes in academic programs and student supports. 


Funding Awards History 

Awards 

For 2008-2009 

For 2009-2010 

For 2010-2011 

For 2011-2012 

Performance 

Performance 

Performance 

Performance 

Districts receiving 
awards 

30 

30 

20 

21 

Total awards 

$1,800,000 

$1,751,960 

$672,495 

$1,168,776 

Dollars awarded per 
point gain 

$31 

$40 

$82 

$84 

Maximum award 

$169,434 

$201,600 

$133,086 

$189,168 

Minimum award 

$4,887 

$2,200 

$3,526 

$5,376 


Due to the unanticipated state budget downturn, the State Board has not been able to honor the 
design principles it adopted for incentive funding. Having been promised new funding to reward 
performance gains, college presidents and their colleagues were not pleased to see their performance 
gains “rewarded” by a loss in overall funding if they were unable to earn back in performance gains 
what they had lost through funding cuts — which no college did, given the magnitude of the funding 
cuts. Further, with the SAI funded through reallocation of base funds, colleges felt a greater sense of 
competition with one another than they had expected to experience. 

State Board Revisions 

The State Board made a number of revisions to the funding principles to try to address the issues that 
arose with the forced use of reallocated base funds. The thrust of the changes was to acknowledge that 
reallocated funds, rather than new funds, would likely be the source of SAI funding in the future and 
to commit to a fair means of reallocation. An analysis by Board staff members confirmed the concerns 
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expressed by college presidents that inequities had resulted from actions to reduce base funding 
on a pro rata basis while allocating SAI funds on the basis of achievement point gains. The Board 
revised the funding principles to give colleges equitable chances to gain back, in SAI funding, the 
dollars they lost in base reallocations. As mentioned, the Board also revised the formula for deter- 
mining each college’s award from the achievement points earned, replacing a single metric of gain 
in total points with a three-part formula using total points, points per student, and completions. 
The rationale for this change was again one of equity — to give every college a fair chance to earn 
awards based on performance. 

Our Assessment of the Principles 

One weakness of the original principles, evident in hindsight, was the reliance on new funds to cre- 
ate the pool of performance funds. This approach was inadvisable for reasons beyond the fact that 
it was unsustainable in a recessionary economy. A reliance on new money leaves in place the incen- 
tives embedded in base funding formulas — incentives widely believed to impede change, as they 
reward enrollment rather than student achievement and institutional performance. A system that 
leaves prevailing incentives in place and creates a separate pool for “performance rewards” sends the 
message that performance only matters at the margins — that colleges are funded mainly to operate, 
irrespective of performance. 

One SAI principle stated that “new funds provide the greatest incentive.” We understand why colleges 
would prefer new funds to reallocated funds, but we are unaware of any evidence to support the claim 
that new funds are more effective in changing institutional behavior than are base funds. Base funds 
have, by all indications, been extremely successful at influencing institutional behavior in the pursuit of 
enrollment growth. It would be reasonable to assume that base funds allocated on the basis of a mix of 
enrollment and performance goals would influence institutional behavior to seek both enrollment and 
success in a more integrated fashion. 

The principle favoring “stable and predictable” funding reflects the view of some college leaders that 
allocating significant amounts of funds on the basis of performance would destabilize colleges and 
make budgets unpredictable. But again, we are unaware of any evidence to support this claim. To the 
contrary, success rates have proven to be stubbornly stable in recent years despite concerted efforts to 
improve them, while enrollments have fluctuated substantially. The justifiable concern about stability 
likely arises from the assumption that performance funds must be held aside and allocated later. But 
performance funds do not have to be withheld. Performance can be incorporated into the overall fund- 
ing formula, based on prior-year data. 

A final concern we have with the State Board principles, even after the adopted revisions, is that 
collectively these principles create a categorical approach to funding performance that is likely to 
produce piecemeal rather than systemic change — contrary to the Board’s own goal. The hundreds of 
interviews we conducted at Washington colleges over the three-year span of our evaluation confirm 
this view. The SAI funds were primarily seen as, and used for, supporting individual initiatives that 
generally did not operate on a large enough scale to produce gains in student achievement overall. 
Interviewees who had not received SAI funds generally felt unaffected by the initiative in terms of its 
impact on their own work. 
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Having been conditioned to expect new money to be used “as a resource for adopting and ex- 
panding practices,” college faculty and staff were not disposed to think of SAI dollars as a means 
to motivate fundamental shifts in institutional priorities toward student achievement. When 
SAI funds had to be used to offset (sometimes only partially) base budget reductions, colleges 
found it difficult to support the new activities they had implemented, further diminishing the 
impact of the SAI on student achievement. 

Fostering Systemic Institutional Improvement 

Impact of Design Principles on Implementation 

The State Board adopted three design principles that relate to improvement of student achievement 
and organizational performance, shown in the box titled “Principles for Systemic Improvement.” The 
SAI was designed to honor these principles in several ways. For one, the metrics were based on research 
by the State Board staff, the Community College Research Center, and others. Many of those we 
interviewed said that they supported the SAI achievement points framework because it was grounded 
in research. Under the SAI, colleges are given data intended to help them assess their performance and 
identify opportunities for improvement. At the same time, colleges have the flexibility to develop 
strategies for improving student achievement according to their particular conditions and priorities. 
The State Board also strongly encouraged colleges to share effective practices, relying on the various ex- 
isting statewide councils of college personnel and on the strong culture of collaboration among colleges 
in the system that was touted by numerous interviewees. 

Principles for Systemic Improvement 

• The initiative leads to improved educational attainment for students, specifically the goal of 
reaching the “tipping point” (of at least a year of college and an occupational certificate) and 
beyond. 

• The initiative allows colleges sufficient flexibility to improve student achievement according to 
their local needs. 

• The initiative results in the identification and implementation of successful practices to improve 
student achievement system-wide. 


In practice, however, the policy has not had the intended impact on college practices and perfor- 
mance. As mentioned, the changes in practice implemented in response to the SAI were generally 
small-scale rather than systemic. And although our quantitative analysis found some evidence of 
increased momentum among students who had already accumulated college credits and made prog- 
ress, progression rates overall do not seem to have changed much since the SAI was introduced. 

State Board Revisions 

In its 20 1 2 revisions, the Board did not make changes to the overriding principles of the SAI. The 
revisions the Board did make in other areas were based on recommendations from a system-wide 
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advisory group that spent more than a year reviewing the policy. This group was organized by the 
statewide association of two-year college presidents and included presidents as well as college rep- 
resentatives from instruction, student services, research and planning, and other areas. We believe 
that the presidents’ substantial efforts in reviewing the S AI policy reflect a commitment to ensuring 
that the policy realizes these principles for improved student success and institutional performance. 
On the basis of the interviews we conducted with presidents, trustees, and others at over half of the 
colleges in the state as part of our field research, we think that this commitment on the part of college 
leaders is strong and will continue. 

Our Assessment of the Principles 

Research on high-performing organizations both in and outside of higher education suggests that to 
substantially improve rates of student progression and completion, colleges cannot rely on discrete 
interventions but must make systemic changes to practice that address all phases of the student experi- 
ence . 9 Yet, where interviewees could identify specific changes in practice implemented in response 
to the S AI, those changes tended to be relatively small pilot efforts focused on particular supports for 
students and faculty rather than systemic reforms of programs and services. 

The research on high performance organizations also indicates that for reforms to be substantial and 
sustained, there needs to be broad engagement of faculty and staff in the institutional improvement 
process. Yet, our interviews indicated that in most colleges, engagement in innovations related to the 
SAI was limited to faculty and staff in pre-college programs and college math, as well as those who were 
involved in efforts directly supported with SAI funds. 

Our field research suggests that the absence of systemic changes is related to the design features of the 
SAI and not a result of any lack of commitment or effort at the colleges. In our interviews, we heard 
frequent references to efforts by colleges to build a “culture of evidence” in which faculty and staff at- 
tempt to change practices based on a review of data on student progression and outcomes. We did see 
indications that such a culture is developing at many colleges, though it is still in its early stages. But 
such efforts have not generally resulted in systemic reforms because the capacity of the colleges to use 
data for improvement is just beginning to develop, and, probably more importantly, the financial incen- 
tives have been too small to elicit college-wide attention to systemic change. 

The experience of the SAI suggests that many colleges need help not only in conducting analyses of 
data on student progression but also in using the results to identify improvement opportunities and in 
designing and monitoring improvement strategies. The efforts by the system’s Research and Planning 
Commission and State Board staff, who used the SAI data to conduct longitudinal analyses of progres- 
sion patterns of student cohorts, provide a promising model for how state agency and college research- 
ers can jointly develop metrics that colleges can use to inform improvements. The State Board has run 
these analyses for every college in the state, providing support that is particularly beneficial for small 
colleges and others that have limited institutional research capacity. 

The State Board has also played a convening role to promote engagement across colleges, working with 
statewide associations of college presidents, academic and student services administrators, institu- 
tional researchers, and other college staff. Initially, the Board was seen as having used this convening 
role effectively in introducing and explaining the SAI, giving updates at various statewide meetings, 
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and providing the SAI data to colleges. However, the Board has only recently played a more active role 
in helping colleges build their capacity for institutional improvement. It is important to note that capac- 
ity-building assistance from the State Board can help, but whether colleges engage in systemic changes 
that lead to improved performance depends even more on there being incentives to make more than 
isolated innovations in practice. 
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Principles to Guide the 

Development of Effective 
Postsecondary Performance 
Funding 


As we stated in the March 2011 policy brief on the S AI, 10 the Washington State Board for Com- 
munity and Technical Colleges deserves high praise for its development and stewardship of the SAL 
In addition, the State Board’s efforts to respond to internal and external evaluations of the initiative 
have been exemplary. We consulted with the Board staff over the course of our three-year evalua- 
tion, and they were exceedingly responsive to the issues raised by our findings. They coordinated an 
internal review of the SAI with the system-wide review group organized by the college presidents. In 
December 2012, the Board adopted changes to address the problems and opportunities for improve- 
ment identified through the review. Given their efforts, it is no surprise that the State Board and its 
staff were the subject of a case study of effective community college coordination and leadership, 
which sought to help other states understand and improve their own coordination function. 11 Here, 
we draw on the Washington experience to help leaders in other states think about how to design 
performance funding systems. 

The SAI offers clear lessons about the process of developing, implementing, and monitoring a perfor- 
mance funding system. Other states have already drawn on a number of those lessons, including that: 

• a performance funding system should be developed through an inclusive, 
informed process in order to get the buy-in of internal and external 
stakeholders; 

• there should be a period of low-stakes implementation to promote learning 
and adjustment; 

• continual communication with internal stakeholders is critical as 
implementation proceeds; and 

• it is helpful to begin with design principles to guide development, 
implementation, and ongoing assessment of the system. 

The initiative also offers lessons about the design of performance funding systems with respect to 
three important goals: 

• to provide timely and valid data to support continuous improvement efforts 
related to performance, 

• to provide motivation through financial incentives for improving 
performance, and 
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• to foster systemic institutional change so that improvements can rise to the 
level and scale that is needed to achieve marked improvements in student 
outcomes. 

We have addressed these three topics in the previous section of this brief, with specific reference to 
the design principles adopted by the Washington State Board for Community and Technical Col- 
leges. In this section, we put forward our own suggestions for principles, drawing on what we have 
learned from Washington and from our other work related to the potential of performance funding to 
transform outcomes for students . 12 Our suggestions are not intended to serve as an exhaustive list of 
principles but rather serve to emphasize some key lessons we have drawn from our work on the SAI. 

Principles for Measurement 

Measure what matters most (and validate that you have done so). 

Some performance funding programs aim to improve a range of outcomes in addition to student 
completion, such as time to completion; production of degrees in science, technology, engineering, 
and mathematics (STEM) fields; and closing of performance gaps. Regardless of the breadth of de- 
sired outcomes, states should carefully measure the outcomes they consider most important, even if 
new measurement approaches are needed. The initial SAI metrics fell short of expectations because, 
as cross-sectional measures, they did not fully capture student progression. Point gains earned by 
colleges did not necessarily reflect progress toward completion by individual students. 

We encourage states that are considering using a system of intermediate outcomes to start by 
measuring the progression of individual students. As we showed with our quantitative analysis 
of the SAI data, such measurement can be done on a year-to-year basis. If states focus only on how 
many students reach certain points, they risk misinterpreting gains in points as improvements 
in student progress. A more general application of this principle is that states should continually 
monitor the validity of the measures they are using as well as their alignment with goals, as goals 
may shift over time. 

One-size metrics do not fit all stakeholders. 

Conventional wisdom suggests that it is best to choose a small set of metrics, but different purposes 
require different sets of metrics. In order to be accountable to the public and to lawmakers, state poli- 
cymakers need metrics that speak to publicly valued outcomes, including the return on the public in- 
vestment in these institutions. These metrics may be the same as those used to tie outcomes to some 
funding. However, to promote institutional improvement, colleges require a different, more detailed 
set of metrics that will help faculty and staff understand which campus practices and policies may 
need to change in order to achieve better results. 

The SAI’s cross-sectional metrics were designed at the outset to better communicate to lawmakers 
and the public about the colleges’ mission and to colleges about the importance of increasing stu- 
dents’ progress through intermediate achievements toward program completion. However, the met- 
rics were not as well designed to provide data to the colleges to inform their improvement efforts, 
leaving many staff members at the colleges unsure why their points went up or down and frustrated 
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at not knowing how to address trends revealed by the data. State and system agencies should work 
with colleges to develop and use metrics that help to inform institutional improvement and also 
align with policymakers’ metrics so that institutions can ultimately see the impact of their reforms in 
the metrics used for public accountability and funding. 

Principles for Funding 

Everyone probably can’t win — but guard against unintended consequences. 

An effective performance funding system aligns funding with desired outcomes, such that a college 
that improves on specified measures gets more funding than it otherwise would. Ideally, high per- 
formers would get more dollars, but in times of constrained budgets, high performers might only get 
a higher share of the budget — still more than they otherwise would. It is tempting to try to eliminate 
the element of competition from a funding model, as the S AI tried to do though an explicit principle 
(see textbox, “Principles for Incentive Funding”), but doing so is neither feasible nor desirable. Sim- 
ply stated, a dollar earned by one college is a dollar not earned by another — and historically, colleges 
have competed for enrollments without questioning the fairness of enrollment-based funding. 

Many performance funding systems accept that there is competition but try to guarantee that no one 
loses by banking on new funds and keeping the stakes small. But these efforts to minimize competi- 
tion ensure that the systems will have minimal impacts. It is vital and feasible to structure a funding 
formula to ensure that colleges are not disadvantaged for serving vulnerable populations and to guard 
against other unintended consequences. However, for fiscal incentives to produce change at the scale 
needed in American higher education, it will not be possible to ensure that everyone wins. A better 
approach is to use fiscal incentives to motivate real change, phase in consequences to allow for learn- 
ing and adjustments, and provide substantive technical assistance for colleges that need it. 

Measure improvements in performance rather than relative performance. 

The performance of colleges is determined in part by the characteristics of the students they serve 
(especially how well prepared they are for college upon entry) and the mix of programs they offer. 
Therefore, although colleges will compete for funding and will inevitably compare their perfor- 
mance to that of others, and although benchmarking performance in particular areas of practice 
can be instructive, a college’s performance is best measured against its own historical baseline. The 
characteristics of a college’s student body and its program offerings can change over time, but mea- 
suring improvements in performance remains a more equitable approach than trying to compare 
the performance of colleges that serve more disadvantaged students with the performance of those 
that serve more advantaged students. It also helps colleges to focus on their own performance and 
what they need to change to improve outcomes for the students they serve. Even within colleges, 
the performance of individual academic programs can best be gauged not by comparing outcomes 
across programs but rather by examining trends over time in the outcome rates for students in 
each program area. For a college’s overall performance rate to improve, efforts need to be made to 
increase rates of program entry and completion across all academic programs, particularly those 
serving larger numbers of students. 
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Performance is a baseline expectation. 

Performance funding in higher education has been hampered by a powerful mental model that has 
crowded out more promising approaches. Under the prevailing view, it is assumed that performance 
should be rewarded after the fact and outside of normal funding. This view implies that colleges 
should be funded to operate irrespective of performance and that other funds — beyond the baseline — 
should be found to reward performance. That way of thinking limits the potential of performance 
funding through its adherence to a number of unhelpful ideas: that performance has to be supported 
with new money, that only small amounts can be used for incentives, and that the principal mission 
of colleges — in which states invest — is to enroll students. 

Those who are skeptical of performance funding claim there is little data to prove its effective- 
ness, but they are looking in the wrong place. They are looking for evidence that a small amount of 
money, unaccompanied by changes in funding priorities, can bring about significant change. The 
strongest evidence that baseline funding incentives work is how well colleges have organized and 
conducted themselves to maximize enrollments or meet enrollment targets. A baseline fund- 
ing formula that valued student progress, completion, and other desired outcomes as much as 
enrollment would likely have a similarly strong effect on college operations and the mindsets of 
higher education leaders. Some states, such as Ohio and Tennessee, are incorporating performance 
elements into the basic funding formulas for their college systems. By doing so, they are sending 
a message that taxpayers are investing in access and success. Incorporating performance elements 
into baseline funding formulas provides colleges with their full allocation in a timely and predict- 
able way, and it allows for larger sums to be allocated on the basis of performance — increasing the 
motivation for systemic rather than piecemeal change. 

Principles for Systemic Institutional Change 

It’s about mission, not rewards. 

We contend that fiscal incentives should be structured as part of a state’s base funding formula to 
colleges. For this approach to work, it needs to be supported by better communications — even a new 
vocabulary — about the purpose of performance funding. SAI funds were presented to colleges as 
“rewards” to be started with “seed money” and continued “as a resource for adopting and expanding 
practices.” By contrast, funding for enrollment has never traditionally been described as a reward. It 
is described and viewed as a resource provided so that colleges can do their jobs. To the extent that 
the conversation about performance funding can shift in focus from rewards and discrete practices to 
base funding and systemic institutional improvement, the power of fiscal incentives may be better 
harnessed to the benefit of student success. 

Performance is everyone’s business. 

Providing the motivation for institutional improvement — through funding formulas and strategic 
communication — is essential, but the process of improvement also requires the expertise of an in- 
stitution’s faculty and staff. Therefore, the data generated by useful measurement frameworks must 
be disseminated widely and used to support continuous improvement and institutional change. 
Faculty and staff must be actively engaged in interpreting the data and designing improvements, and 
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college leaders must support them in this work. Just as performance improvement should become 
integral to the mission, funding, and expectations of institutions, so should institutional research 
be embraced as a core college function. Some institutions, particularly small colleges and those with 
fewer resources, often lack sufficient capacity to do effective institutional research. State college and 
university systems with system-wide unit record data can help by using these data to conduct lon- 
gitudinal analyses of student progression patterns for all institutions in their purview. By doing so, 
they can help to spread a common way of looking at problems and evidence-based decision making 
throughout entire state systems. 

Observations 

For states searching for ways to improve the returns of their investments in higher education, 
performance funding is not a panacea, but we maintain that it can and should be an important 
component of state policy. State leaders should not be deterred from considering the benefits of 
performance funding by the paucity of direct evidence of its effectiveness or by the opposition 
of some stakeholders, as performance funding has not yet been given a fair chance because we 
are still learning how to do it better. The experience of the Washington community and techni- 
cal college system has revealed much about what aspects of performance funding can work and 
what might be changed to make it work better. Some states are already benefiting from these les- 
sons. An important aim of this policy brief is to show other states that there is a reason to keep 
working to improve how colleges are funded, and if we can learn from experience, performance 
funding holds promise to improve student success. 
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Endnotes 

1 . For a review of the literature on the impacts of state performance funding policy on higher 
education systems, see Dougherty and Reddy (2012). 

2. For details about the specific features of the SAI, see Shulockand Jenkins (2011, pp. 4-7). 

3 . For an overview of the weaknesses of performance funding and the responses embodied in 
new approaches, see Shulock (2011). 

4. The Washington State Board for Community and Technical Colleges in December 2012 
adopted revisions to the SAI points framework. The points shown here are based on the 
original framework adopted in 2007. 

5. Findings from the three-year evaluation are available in Jenkins, Moore, Wachen, and 
Shulock (2012). 

6. Shulock and Jenkins (2011). 

7. See Washington State Board of Community and Technical Colleges (2012, Tab 7) for details 
of the State Board’s actions to revise the S AI . 

8. Washington State Board of Community and Technical Colleges (2009, Tab 8). 

9. Jenkins (2011). 

10. Shulockand Jenkins (2011). 

11. Kirlin and Shulock (2012). 

12. Shulock (2011). 
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